Optimal Training Algorithms and their Relation to Backpropagation

We derive global H 1 optimal training algorithms for neural networks. These algorithms guarantee the smallest possible prediction error energy over all possible disturbances of xed energy, and are therefore robust with respect to model uncertainties and lack of statistical information on the exogenous signals. The ensuing es-timators are innnite-dimensional, in the sense that updating the weight vector estimate requires knowledge of all previous weight esimates. A certain nite-dimensional approximation to these es-timators is the backpropagation algorithm. This explains the local H 1 optimality of backpropagation that has been previously demonstrated.